System and method for event prevention and prediction

ABSTRACT

A computer-implemented method is provided, comprising, based on information of a subject positioned on an object in an environment, generating a data stream; for the data stream, extracting features associated with a movement of the subject with respect to the object or the environment, wherein the movement is represented by spatio-temporal features extracted from sensors; generating a prediction associated with a likelihood of the movement based on the extracted features, and a risk profile of the movement based on a plurality of fall risk factors; and applying the prediction and the risk profile to a rule base to perform an action.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationNo. 62/881,665, filed Aug. 1, 2019, the contents of which areincorporated herein by reference.

FIELD

Aspects of the example implementations relate to methods, systems anduser experiences associated with prevention and prediction of an eventby a combination of action forecasting, electronic health records andindividual event risk profile.

RELATED ART

In the related art, a person may fall from a bed, chair or other object.Such a fall is a significant problem that may cause injury to theperson, such as minor bruises, disability or death. In an in-patient orhospital environment, falls and associated injuries are some of theleading conditions that are acquired by patients during their stay inthe hospital. It is estimated that there are more than 1 millionpatients who fall in hospitals annually in the United States, whichaccounts for 85% of hospital acquired conditions. Further, 29 to 55% ofpatient falls result in injury, and the cost associated with fallinjuries is over $30 billion a year. The related art approaches to falldetection may place significant financial, emotional and legal pressureon medical facilities such as hospitals, as well as their staff

Related art approaches to address the problem of falling have focused onthe event detection approaches, such as use of bed exit alarms or padson the bed or floor, to trigger alerts after a patient has fallen out ofa bed. The related art approaches analyze sensor data to detect fallsafter they have occurred. However, in addition to the sensor data, thereare other multifaceted factors, such as demographics, health conditionsand ambient or surrounding conditions that may contribute to a fall.

FIGS. 1(a)-1(c) illustrate various related art approaches. Some relatedart approaches are pressure based, such that in response to an absenceor presence of pressure on the pad, an alarm is triggered. For example,FIG. 1(a) illustrates a pressure sensitive pad 101 positioned below apatient sleeping on a bed, such that when pressure is applied to thepressure sensitive pad 101, an alarm may be triggered. However, thisrelated art approach may have various problems and disadvantages. Forexample, this related art approach is reactive, and may not be activatedsufficiently quickly prior to the patient being out of the bed. Forexample, but not by way of limitation, the pressure based related artapproach may result in frequent false alarms, due to weight differencesand non-falling movement of a patient, as well as the issue of impropertiming of activation as explained above. Thus, pressure pads may not beaccurate to detect a patient getting out of bed.

Alternatively, a patient may manually trigger an alarm upon or afterfalling. FIG. 1(b) illustrates a bed exit alarm 103 that is positionedon the frame of a bed in which a patient may be sleeping. Otherapproaches include use of sitters with the patient, or bed restraintsthat prevent movement of the patient altogether, as shown in FIG. 1(c)as bed restraint 105.

Additionally, in the related art nurses must devote additional resourcesto patients with a higher fall risk. The assessment of fall risk may bebased on related art risk assessment tools, such as the Morse Fall Scaleor motor tests. However, this related art approach has various problemsand disadvantages. For example, but not by way of limitation, these riskassessments are not reliable, as virtually all patients in the hospitalwould be rated as having a high fall risk; thus, it is not a useful toolto allocate resources to certain patients with higher fall risk.Further, fall risks may be related not only to mobility, but otherfactors that are not covered in the related art risk assessment tests.Those other factors may include surrounding environment, medicalprocess, and the like, for example.

Additionally, even if nurses taking related art approaches devoteadditional resources, the patient may not remember or be able to callfor the nurse when intervention is needed. Moreover, because thedifficulty of risk assessment presents challenges with respect toplanning as to when a patient might need intervention, related artapproaches make it difficult to prioritize patient care, anddifferentiate which patient may need assistance.

Further, some related art approaches may use motion history imagesequences as visual features and in a Markov models for activityclassification, motion history image requires manually selecting thewanted motion and is sensitive to illumination or slight trembling ofthe camera, and may miss subtle human posture changes. Such related artapproaches only focus on in—bed activity, and does not use contextfeatures for analysis of a risk of a fall. The related art approach alsoapplies segmented motion history, instead of the dense trajectory ordeep learning approaches described here.

SUMMARY

Aspects of non-limiting example implementations related prevention offalls, and more specifically, to anticipating falls before they happenin order to potentially prevent the falls, instead of only reacting tofalls after they have been detected.

According to an aspect of the example implementations, acomputer-implemented method is provided for, based on information of asubject positioned on an object in an environment, generating a datastream; for the data stream, extracting features associated with amovement of the subject with respect to the object or the environment,wherein the movement is represented by spatio-temporal featuresextracted from sensors; generating a prediction associated with alikelihood of the movement based on the extracted features, and a riskprofile of the movement based on a plurality of fall risk factors; andapplying the prediction and the risk profile to a rule base to performan action.

Example implementations may also include a non-transitory computerreadable medium having a storage and processor, the processor capable ofexecuting instructions for prevention and prediction of an event byaction forecasting, electronic health records and individual event riskprofile.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIGS. 1(a)-1(c) illustrate various related art approaches.

FIG. 2 illustrates various aspects of a subject and a room according toexample implementations.

FIG. 3 illustrates various aspects of the system according to someexample implementations.

FIG. 4 illustrates a workflow associated with a visual system orpredicting an action or an event according to some exampleimplementations.

FIG. 5 illustrates a pictorial illustration of a workflow associatedwith a visual system or predicting an action or an event according tosome example implementations.

FIGS. 6(a)-6(c) illustrate dense sampling of local features according tosome example implementations.

FIG. 7 illustrates extraction of prediction instances according to someexample implementations.

FIG. 8 illustrates an example process for some example implementations.

FIG. 9 illustrates an example computing environment with an examplecomputer device suitable for use in some example implementations.

FIG. 10 shows an example environment suitable for some exampleimplementations.

DETAILED DESCRIPTION

The following detailed description provides further details of thefigures and example implementations of the present application.Reference numerals and descriptions of redundant elements betweenfigures are omitted for clarity. Terms used throughout the descriptionare provided as examples and are not intended to be limiting.

Aspects of the example implementations are directed to systems andmethods associated with prevention of an event prior to the occurrenceof the event. For example, the event may be a movement of a subject(e.g., person) relative to an object (e.g., environment), and morespecifically, a fall of a person (e.g., patient) from a bed or a chair,such as a subject in a hospital or a resident of a nursing home,subsequently referred to as the subject or subject. For example, but notby way of limitation, the foregoing example implementations may beapplied in a non-hospital setting, such as in a residence of thesubject, to address conditions such as sleepwalking or other abnormalmovement in sleep. Further, the example implementation may beincorporated with other types of sensors that may predict and assessrisk with respect to other types of conditions, such as sleep apnea,sudden infant syndrome (SIDS), or other conditions. In a manner similarto that explained above, with respect to control of a hospital bed, theexample implementations may control other medical devices, such asintravenous feed, oxygen supply, breathing support, etc.

A vision based method is provided to predict if a subject is attemptingto exit a safe position, based on motion trajectories and deep recurrentneural networks (RNNs). Further, dynamic and subject specific fall riskprofiling is provided based on intrinsic and external factors, asexplained below. Further, the combination of the predictive model ofwhether a subject is about to exit a position in a bed combined with therisk profiling is used to generate signaling strategies that areprovided for prevention of falls that is customized or tailored based ondiffering severity of falling in different risk situations.

More specifically, example implementations include systems and methodsfor fall prevention, such as fall predictive analysis and signals forprevention. Information received from sensors such as video cameras maybe used to forecast the risk of injury for a subject, and maycontinuously estimate the likelihood of a subject moving from a safeposition, such as in a bed or a chair.

For example, the sensors may be nonintrusive vision based sensors in asurrounding of the subject that may monitor activities of the subject.Color, grayscale, depth and other aspects of videos may be provided.Further, sensors may be provided to capture passive infrared, microwave,ultrasound or other medium to detect motion of a subject.

Further, it is noted that the sensors do not just sense activity at theimmediate location of the subject himself, but also sense the entireenvironment of the subject, including but not limited to the subject,the safe location of the subject such as in a bed or a chair, and theambient surroundings of the room in which the subject is located.

This likelihood is dynamic, and may differ from subject to subject.Moreover, the likelihood will generally be lower when a subject is in asafe position, such as lying down in a bed or sitting in a chair, andincreases as the subject moves from the safe position and attempts totransition out of the safe position. By anticipating a fall before thefall actually happens, it may be possible to prevent the fall.

The system may continuously update by monitoring risk factors based oninformation received from health records of subjects, behavior patterns,vital signs and surrounding conditions, for example. A ranking may beprovided based on a severity of signaling for providing recommendationsas to a signaling strategy to be executed in order to prevent the fall.As a result, instead of detecting a fall after it has occurred, the fallis predicted prior to occurrence.

The example implementations employ a vision based approach that predictswhether a subject is attempting to exit a safe position. Spatio-temporalvisual features (e.g., dense trajectories) are used to extract localatomic action patterns from data streams (e.g., image streams) of amoving body. Further, Fisher vectors are used to encode features, andneural networks are used for activity prediction associated with theenvironment, such as the bed or chair.

Further, the example implementations employ dynamic and subject specificrisk profiling, which is associated with a risk of a fall. For example,activity histories, intrinsic factors and external factors may be fusedto provide personalized risk analysis associated with a fall. Multiplerisk factors associated with a fall may be encoded into variables, and afall risk may be computed using regression. By using an analysis of thebehavioral patterns of a subject, a predictive model may be generated topredict a next action of the subject. For example, but not by way oflimitation, if a subject is lying down and suddenly sits up, andlikelihood of the subject attempting to move out of the bed can becalculated. Based on this calculation, a signal can be provided to thesubject or another entity in a position to attempt to stop the fallbefore it happens. Thus, a signal that has one of a plurality of levelsof likelihood is generated and provided that may be used to prevent oravoid falls.

Additionally, fall prevention signaling strategies are providedaccording to the example implementations. For example, a severity ofintervention may be determined as well as a timing, and how tointervene, by combining a subject's fall risk and a likelihood of thesubject exiting a safe position.

Aspects of the example implementation are directed to prediction of anevent, and more specifically, predicting a subject exiting, or gettingout of, a bed or a chair. The example implementations are directed toearly prediction and risk assessment, followed by timely action. Theapproaches described here in combine action forecasting, electronichealth records and individual fall risk profile.

FIG. 2 illustrates the proposed system 200 according to the exampleimplementations. In an environment 201, an object 203, such as a bed orchair is provided. A subject 205, is in a resting position on the object203. Further, a motion sensor 207 and a camera sensor 209 are provided.The motion sensor 207 may include, but is not limited to a type ofmotion sensor that is passive infrared microwave, or ultrasound, or acombination thereof. The camera sensor 209 may be selected from one ormore types, including but not limited to grayscale, RGB (red greenblue), night vision, depth and/or thermal. The motion sensor 207 and thecamera sensor 209, may be placed in a combined location, such as on aceiling or a wall of a room, or may be placed in separate locations, andmay be positioned or selected to provide the necessary information in aprivacy protecting manner, such as use of depth camera or motion sensor,for example. Other privacy preserving approaches may be provided,including but not limited to use of a motion sensor instead of a camera,anonymization of information sensed or collected with respect to thesubject, the environment, or the like.

According to the example implementations, and more specifically, byincorporating the data received from the one or more sensors with theelectronic data such as hospital information, into an automatic learningsystem, deep neural networks provide vision based action forecasting, topredict a likelihood P of the subject exiting a safe position. Further,intrinsic, individual and contextual features are fused, to estimate arisk R of falling.

For example, the likelihood P_(t) of a subject exiting the sameposition, and a fall risk R_(t) associated with the subject falling attime t may be calculated. For example, but not by way of limitation ifthe subject is lying in a bed, P_(t) represents a likelihood of asubject exiting the bed, and R_(t) represents a continuous riskprofiling as to a risk that the subject will fall. When R_(t) is high,such as above a prescribed threshold, the subject or others charged withtaking care of the subject should be provided with a high recall, so asto be able to advance to assist high risk subjects. On the other hand,when R_(t) is low, due to increased precision associated with theexample implementations, false alarms may be avoided.

Additionally, P and R are combined to determine a severity of a fallprevention intervention strategy to be applied. For example, a signalmay be generated and emitted based on a level of severity according tothe following rule base:

If P is low and R is low, do not signal

If P is low and R is high, signal the subject to suggest request forassistance

If P is high and R is low, signal the subject to follow a safetyprocedure

If P is high and R is high, signal the subject to remain in a safeposition. and also signal staff directly for immediate assistance.

FIG. 3 illustrates a system overview according to an exampleimplementation. As shown in the system overview 300, a plurality ofinputs are provided from the camera and/or motion sensors discussedabove.

More specifically, as shown at 301, spatio-temporal visual features areprovided from sensors such as cameras. For example, but not by way oflimitation, the inputs from the sensors at 301 may include, but are notlimited to video streams 311, for which spatio-temporal visual features313 are generated, such as dense trajectories, but not limited thereto.The cameras sense not only the subject, or even the bed, but the entireenvironment in which the subject is located, including but not limitedto the ambient environment of the room itself.

As shown at 303, information may be provided from electronic records,such as historical events and health records. For example, but not byway of limitation, the historical events include, but are not limitedto, a time of a last event, such as toilet visit, at 319, a subjectaction, such as an alarm being pressed at 321, or the like. Healthrecords may include, but are not limited to, information associated withsubject vital signs, such as blood pressure at 315, and informationassociated with past relevant events, such as numbers of previous fallsat 317.

The sensed information from 301 and the electronic records informationfrom 303 are provided as inputs at 323 to the recurrent neural network(RNN). More specifically, the present example implementation may includea long short-term memory (LSTM) RNN or deep convolutional neural network(CNN), including a plurality of frames 325. In this exampleimplementation an approach having two layers at 327 is provided.

Table 1 shows examples of sensed information 301 and electronic recordsinformation 303 that may be input into the example implementations. Therisk factors are divided into three main categories: person specific,environmental and behavior routines. Person specific risk factorsinclude information that is associated with the subject, such as vitalsigns, fall history, medical information, physical information ordemographic information. Environmental fall risk factors includeinformation associated with ambient surroundings, such as location,units such as room type, presence of roommates, light intensity andtemperature, and time information. Behavior risk factors includeinformation associated with behavior routines, such as requesting ofassistant, toileting (e.g., continuous activity), sleep patterninformation (e.g., categorical activity) and medical process. Moreover,the risk factor variables may be characterized as continuous or discretevariables.

In addition to having the information collected for the current medicalincident, information may be collected in advance, such as based onprior medical incidents, or other activity occurring at home or in ahospital, for example. According to some example implementations, thesame sensor may be provided in the home and in the hospital, andinformation that is received at each setting may be calibrated for thelocal environment.

TABLE 1 Fall Risk Factors Person specific vital signs x₁: continuousscale of deviations in blood pressure fall history x₂: number of fallsin the past medical x₃: severity levels of diagnosed neurologicaldisorders (i.e., Parkinson, Parkinsonian disorder), Diabetes, Depressionor Incontinence physical x₄: BMI x₅: severity levels of impairedmobility x₆: severity levels of impaired visual demographic x₇:log(age - 65) x₈: 1 if female gender Environmental location x₉:normalized distance to the nursing station units x₁₀: categorical type[1-medical-surgical units; 2-intensive care patients] x₁₁: number ifpresence of roommates x₁₂: light intensity x₁₃: temperature time x₁₄: 1if night shift x₁₅: number of days in the hospital x₁₆: number of daysafter major surgery (i.e., knee replacement) Behavior assistant x₁₇:normalized frequency bed alarm routines request pressed toileting x₁₈:time to last toilet visit sleep x₁₉: deviations of sleep patternsmedical x₂₀: number of in-taking medications process intake >4 x₂₁:nurse staffing [1-low; 2-medium; 3- high]

The risk factors described in Table 1 are only examples, and other riskfactors may be included or substituted for these risk factors, as wouldbe understood by those skilled in the art. As noted above, related artapproaches do not consider environmental risk factors.

In addition to a measurement of the current state of the foregoing riskfactors, additional example implementations may include longitudinalmeasurement. For example, a risk assessment may be taken over time for asubject, and environment and/or behavior routines. The change in therisk assessment over time may be used to predict future changes in therisk assessment. In one example implementation, risk factors, such asthose shown in Table 1, may be taken for a subject in time intervals,such as on a yearly basis, every five years, etc. The results may beanalyzed in a longitudinal manner, to provide a forecast of future risk.

Further, in addition to changes in the risk factors that may be specificto the person, risk factors may also change with respect to theenvironment. For example, but not by way of limitation, different roomswithin a hospital may have different risk factors for the same subjectand the same behavior routines, or different hospitals may havedifferent risk assessments for the same subject and the same behaviorroutines, in the same department of the hospital. These changes may bedue to resource allocation, staffing, allocation of rooms, layout ofhospital, etc. Such factors may be incorporated into the determinationof the risk factors, as explained herein.

An output 305 of the LSTM RNN includes a dense layer 329; the LSTMlayers are combined with the dense layer to predict a likelihood of theevent, such as getting out of bed at various time (e.g., timestamps). Inthe present illustrative example implementation, timestamps of 1, 5, andk are represented at 331, 333 and 335, respectively.

The outputs illustrated at 305 providing the predicted likelihood ofgetting out of bed are combined to estimate a current fall risk.Optionally, an attention—based LSTM may be used. More specifically, thisapproach may learn the importance of features during temporalrelationship mining. Alternatively, other example implementations oralternatives may also be used, using alternative sequence models, suchas gated recurrent units (GRU) and temporal convolution nets (TCN).

FIG. 4 illustrates a workflow according to the example implementations.More specifically, a workflow 400 associated with video monitoringcomponents for action forecasting is provided. The output is acontinuous scale P∈[0,1] associated with a likelihood of a subjectmoving out of a safe position, such as a subject getting out of bed, ata current time.

At 401, sensing devices, such as camera sensors, are provided to sensethe environment. For example, and as discussed above, cameras may beemployed, including but not limited RGB, RGB-D, thermal and night visioncameras. For example, but not by way of limitation, a depth camera or athermal camera may be used to segment the subject in the foreground. At403, movement of the subject on the bed or chair is streamed, as well asthe human movements that may occur in the environment surrounding thebed, such as the room in which the subject is located.

At 405, a feature extraction operation is performed, such that featuresare extracted from a sequence of frames. For example, a two streamconvolutional neural network (CNN) architecture may be employed to stackimages and optical flow sequences, to represent frame level features.The spatial data from the image sequences includes appearanceinformation, and the temporal data from the optical flow is associatedwith motion information. Alternatively, features such as skeleton jointsmay be used as feature representations.

According to the example implementations, a window of consecutive framesmay be processed to extract features representative of the spatialtemporal dynamics of movement patterns. For example, but not by way oflimitation, such movement patterns may include a subject rolling in bed,and movements associated with a subject sitting up or lying down.According to one example implementation, densely sampled localspatio-temporal features known as dense trajectories may be used torepresent action dynamics. Such dense sampling of local features mayoutperform sparse spatio—temporal interest points.

Accordingly, the example implementations are directed to use of densesampling approaches and extracted local features along the densetrajectories. Accordingly, features that are highly relevant to humanactions may be encoded, without a requirement for backgroundsegmentation. Approaches that may be used in the example implementationsthat may be used in the example implementations include, but are notlimited to, histogram of oriented gradients (HOG), histogram of opticalflow (HOF) and motion boundary histogram (MBH). However, otherapproaches may be substituted for these approaches, as would beunderstood by those skilled in the art.

At 407, action features are learned. For example, video training data ofsubjects moving in beds, sitting on chairs and walking may be used suchthat the system learns codebooks for feature encoding.

At 409, feature vector sequences are encoded. Encoding the featuredescriptors may include a combination of different feature types or useof a subset of the features. During training, action features areclustered, for example using Gaussian mixture models (GMMs). Duringtesting, features extracted from a window of consecutive frame may beencoded as Fisher Vectors that can learn the distributions ofdescriptors, and may thus be less sensitive with the occurrencefrequency of each word, as well as the encoding additional informationassociated with a distance of each descriptor from the center. Accordingto an alternate example implementation, the features may be representedusing bag of visual words (BOV) soft BOV, or hashing.

At 411, the information is fed to the forecasting model, such as an RNNforecasting model. Further details of the forecasting model arediscussed in greater detail below.

FIG. 5 provides a pictorial visualization 500 of the workflow of thevideo monitoring for action forecasting according to the exampleimplementations. A goal of forecasting the human actions with respect toa potential event or action is to predict the action in time, given acontinuous video stream. The video stream may include multiplesequential actions, such as routine activities that contain multipleactions that may have subtle differences from one another, and follow aprescribed sequence. For example, but not by way of limitation, thedifference between a sleep pattern of a subject in normal sleep and asubject in abnormal sleep having a higher fall risk may only have subtledifferences in terms of the subject movement, relative to the overallmovement during a sleep event.

The present example implementations use dense trajectories to extractlocal spatial—temporal patterns, instead of frame level representations.A Gaussian Mixture Model (GMM) is used to build a code book for thedescriptors of the dense trajectories in each of the feature categories(e.g., trajectory, HOG, HOF and MBH). A short sequence of the video isrepresented using Fisher vectors from the trained codebook, followed bya combination of short range action feature representations withhigh-level temporal models, such as TCN and LSTM, to learn long-rangeaction dependencies. In the case of the TCN model, a temporalconvolutional network consists of repeated blocks of convolutions,followed by nonlinear activations.

In the case of an LSTM model, the sequential information of the inputdata is used, and current sub sequences are processed, given informationextracted from previous sub sequences with the use of memory cells.Thus, LSTM can learn both short-term and long-term dependency patternsfrom input features. The present example implementation may include abidirectional LSTM that comprises two reversed unidirectional LSTMs. Thememory units may be followed by a time distributed, dense layer, theresults of which may be fed into an output softmax layer for prediction.

For example, at 501, use of a depth camera or thermal cameras to segmenta foreground human subject, and provide streaming of human movements ona bed or chair is illustrated. At 503, generation of dense trajectoriesfor use in feature extraction is disclosed. Further, at 505, the variousdense sampling approaches are disclosed, which result in learning ofaction features, such as using GMM training action codebooks at 507. At509, encoding of extracted features as Fisher vectors is disclosed, andat 511, an output action likelihood is disclosed.

FIGS. 6(a)-6(c) illustrates an example implementation that employs adense trajectory approach. As shown in FIG. 6(a), a person is in asitting position, and interest points are sampled at a greater densityaround the sitting person. As can be seen in FIG. 6(b) at 603, thesampled interest points can show movement as a different color, textureor shape of the points; because of the greater density of samplingaround the sitting person, more granularity is provided for themovement. Further, as shown in FIG. 6(c) at 605, the densely sampledpoints of interest show movement of the person out of the chair andwalking away.

FIG. 7 illustrates model learning and collection of training instancesfor predicting an action likelihood according to an exampleimplementation, at 700. Training instances of action forecasting are setup as (L, k) timestamps for action classification tasks. At 701, 703,705 and 707, for example, sensed information, such as rolling on a bedor sitting up, are shown as frames. Further, classifiers may classify,based on training data, whether sleep is normal, such as may be found insleep patterns or historical information associated with sleep patternsfor the subject or other similar subjects in similar situations, or onthe other hand, is not considered to be normal sleep activity, which maybe indicative of a risk of falling.

This information represents sequences 715, for feature extractions.Elements 709, 711 and 713 represent upcoming actions or events, atfuture timestamps, such as sitting up, getting out, etc. The tasks maypredict an event or action, such as whether a subject will move from asafe position in a bed or a chair, in the upcoming k timestamps, basedon the learned long-term pattern, as well as recent observations madeduring the prior L frames.

The input to the predictive models is a time series of encoded featurevectors, as explained above, and more particularly as provided at 411 ofFIG. 4. The neural network approach includes use of LSTM, a variant ofRNNs, the model long-term dependencies, and to provide predictions basedon the input. As shown above in FIG. 3 at 327, one exampleimplementation includes two LSTM layers that are combined with one denselayer, to predict a likelihood of a subject moving from a safe positionin the bed, at the subsequent 1, 5 and n timestamps, shown as 717, 719and 721, respectively; the outputs of these predictions are combined, toprovide an estimate of a current fall risk.

As an alternate example implementation, attention—based LSTM may beused, to learn the importance of features during temporal relationshipmining. Other example implementations may use other sequence models,including but not limited to gated current units (GRU) and temporalconvolution nets (TCN).

More specifically, for consecutive M frames from a video subsequenceS={f_(t)}_(t=1) ^(M), X represents the visual feature representationsthat are input to a prediction model. Given the input X_(t) for timestep T, the likelihood of action is predicted at P_(t). According to anexample implementation, if the softmax activation function is used inthe output layer of a deep neural network, the output likelihood ofaction P_(t) may be obtained by taking the probabilities of the currentinput X_(t) that belongs to an existing action.

Accordingly, the present example implementations provide use of densetrajectories to extract local atomic action patterns from image streamsof a moving human body, as well as Fisher vectors to encode features andneural networks to predict events and activity associated with thesubject in the bed or chair, or fall prevention.

In addition to the foregoing prediction information associated with thepredictive models, the example implementations also include dynamic riskprofiling specific to a subject, for each timestamp. These exampleimplementations may provide a high recall or high risk subjects.

As noted above, Table 1 discloses risk factors associated with a fall.Some factors may be provided based on existing medical research andhospital reports. Further, a subject may have different combinations ofrisk factors associated with a fall, and the fall risks may change overtime, including during the subject's stay in the hospital. Further, therisk may vary across different medical units, and may also vary acrosshospitals.

For example, intrinsic subject features, such as age over 65 years,impaired gait, visual impairment, or a low biomass index (BMI) may bemore likely to lead to falls. Additionally, the risk of a fall may begreater for a subject having a history of falls and neurologicaldisorders. Characteristics of the subject room may also impact the riskof subject fall and injury. For example, surgical subjects may have ahigher risk than intensive care subjects, and subjects in single roomsmay have a greater fall risk, due to staffing limitations. Further, timeof day may also be an important factor, as injury may be more frequentduring a night shift, and older subjects may be more likely to fall onthe first day after moving to a new room.

Behavior routines may also impact risk of fall. For example, but not byway of limitation, a subject who has used a call button multiple timesto indicate a need to move from the safe position in a bed, such as togo to the bathroom, may have a higher probability or likelihood ofmoving out of the bed, while other subjects may press the call buttonwithout an actual need to move out of the bed. Individual subject vitalassessments that may affect fall risk after a subject leaves the bed mayinclude blood pressure, medication or recent injuries. For example, adecrease in blood pressure may indicate orthostatic hypertension, whichis associated with a higher fall risk, and taking more than fourmedications may increase the fall risk.

Example implementations provide a profile of a risk of falling based onthe input, including those included in Table 1, and the calculation ofrisk probability. In other words, risk factors may be provided based onmedical research, hospital reports or the like, and data may be providedby sensors, such that the values of each of the fall risk factors thatare obtained include vital signs, electronic health record collection orthe like, are collected and encoded, as either continuous or categoricalfeatures. For example, but not by way of limitation, vital signs may besensed automatically or collected by hospital staff.

According to one example implementation, if a data set is available froma hospital, a regression method may be applied to learn a feature weightb, and to estimate a fall risk R∈[0,1]. Alternatively, hospital staffmay provide a manual indication of an importance of variables collectedby sensors. As noted above, the sensors not only look at the immediatelocation of the subject at the bed, but the overall environment, andthus are configured to sense additional information that may be used asrisk factors. Input risk variables may be encoded as either continuousor categorical features as explained above, and a normalization (e.g.,scaling the risk score between zero and 1, in a manner that permitsstandardization and comparison across medical incidents, subjects, etc.)may be applied on each feature dimension. For the list of N risk factorsx_(i), such as those shown in Table 1, for i∈[1,N], a fall risk R_(t) iscalculated at time t for each subject, as a logistic regression of inputrisk factors, as shown in equations (1) and (2) as follows:

Patient's fall risk score S_(t)=b₀+b_(lowbloodpressure)x₁+b_(#falls)x₂+.. . b_(N)x_(N)   (1)

Patient's fall risk R_(t)=1/(1+exp^(−s) ^(t) )   (2)

Based on the foregoing likelihood of moving out of the safe position P,combined with a risk of falling R, fall prevention signaling strategiesmay be adapted. Table 2 provides an example of fall prevention protocolsdepending on values of P and R, according to the exampleimplementations.

TABLE 2 Fall Prevention Protocols High fall risk R Low fall risk R Highexiting notify staff; warn patients; reminders of safety likelihood Plower bed height; check existing procedure bedrails are up Low exitingreminders of requesting likelihood P assistance

For example, a rule-based approach may be implemented as follows. If Pis below a threshold and R is below a threshold, a signal may not beprovided, as the probability of moving from the safe position as well asthe risk of falling are below thresholds.

In another scenario, if P is below the threshold and R is above thethreshold, a signal may be provided to the subject, recommending thatthe subject request assistance. This situation would be indicative of ahigh fall risk, but the subject not being in a high probability ofmoving out of the safe position.

Additionally, if P is above a threshold and R is below a threshold, thesubject may be provided with a single to follow a safety procedure. Thiscondition may be associated with a high probability of the subjectmoving from the safe position, but a low risk of falling for thatsubject.

Further, if P is above a threshold and R is also above a threshold, asignal may be provided to the subject to remain in the safe position,and also, a signal may be provided to staff, to directly provideimmediate assistance.

The foregoing scenarios are merely examples, and are not intended to belimiting. Other examples may also be implemented, with variations aswould be understood by those skilled in the art. For example, thesignals emitted in the case of a certain scenario may vary, depending onthe local hospital protocol, subject preference, or other basis.Further, threshold for P and R may vary. Further, protocols may not belimited to reminders and staff notifications, but may be integrated intoother aspects of subject movement, such as rehabilitation, physicaltherapy or the like.

FIG. 8 illustrates an example process 800 according to the exampleimplementations. The example process 800 may be performed on one or moredevices, as explained herein.

At 801, information is sensed, such as by a camera or motion sensor asexplained above. Further, sensed information is received, and a streamis generated. The stream is indicative of the movement of an object on asurface, such as a subject on a bed or a chair or floor. Structures andtechniques associated with the sensing of the information and thegeneration of the stream are described above with respect to the exampleimplementations.

At 803, extraction of features from a sequence of frames is performed,and action features are learned. For example, but not by way oflimitation, the use of approaches such as dense trajectories may beemployed, along with additional techniques and implementations asdescribed above.

At 805, feature vector sequences are encoded, such as based on GMMtraining action codebooks, to encode a window of consecutive frames asFisher vectors, for example. The encoded feature vector sequences arefed into the forecasting model, which is described above respect to FIG.3.

At 807, the values of P and R, associated with the probability orlikelihood of a subject getting out of a bed or chair, and a risk offall are calculated. The calculation of P and R is based on thepredictive model and risk profiling approaches as described above.

At 809, the values of P and R are assessed to determine whether they arerelatively high or low. For example, the values of P and/or R may becompared to a threshold. Based on the assessment, a rule-based may beapplied, and one or more protocols may be selected, such as those shownin Table 2 above.

At 811, an action is performed based on the selected protocol. Forexample, but not by way of limitation the action may include providing asignal to the subject, providing a signal to an external resource suchas a third party such as a caretaker or staff, or executing an action tobe automatically performed with respect to a device associated with theuser, such as switching a position of a bed or protective equipment,placing fall mitigation equipment or padding in place, activating analarm, unlocking the door to the room, or other activity as may beunderstood to be an action associated with implementing the protocolbased on values of P and R, indicative of a prediction of a subjectexiting a safe position in a bed or chair, combined with a risk offalling, such as noted in equations (1) and (2) above.

With respect to the actions that may be taken, in addition to theforegoing examples, additional non-limiting examples may be employed.For example, but not by way of limitation, the action may not be limitedto the signal provided to the subject, or a third party such as acaretaker or staff to implement immediate action to prevent the fall.Additional actions may include providing aggregated risk profiles forplural subjects, such as in a hospital department, a hospital floor, orthe entire hospital, in a privacy preserving manner, and based on theaggregated risk profiles, re-allocating resources within a hospital,department or other organizational entity. According to one exampleimplementation, if a fall risk is substantially higher at a certain timeof day or night, additional resources may be allocated, or subjects maybe moved, to address those risk factors.

Further nonlimiting examples of actions that may be performed based on aselected protocol may include, but are not limited to, sending a thirdparty to intervene and prevent the fall, activating lights, such as forfloor guidance, or a sign for the subject, and providing warnings orrecommendations by way of a speaker in the room or on a mobile device ofthe subject; the information provided may be prerecorded messages, lightsignals, audio messages or other communications as would be understoodby those skilled in the art.

As also explained above, the actions may involve controlling devices, soas to reduce the risk associated with a fall by the subject. Forexample, as explained above, features associated with a bed, such as anautomated or motorized rail or other guard, for padding on the floor, orother support structures, may guide the subject to get out of the bed ina manner that reduces the risk of a fall, or even increase thedifficulty of inadvertently getting out of the bed, such as by changinga position, angle, orientation of the bed, moving the bed itself, ormoving other objects within the room.

In some example implementations, the system may interface with a sensingor detection system, or by way of the sensors or cameras describedherein, determine whether a subject has inadvertently not placed aprotective feature of the bed in its proper position. For example, arail detection system to detect when a rail is not in the engagedposition while a subject is sleeping, possibly do to a subjectinadvertently not placing the guard in the appropriate position.

In the foregoing example implementation, the operations are performed atan online mobile application associated with a user. For example, aclient device may include a camera sensor, or communicate directly withthe sensor in the room, and some or all of the processing may beperformed locally on the online mobile application. However, the exampleimplementations are not limited thereto, and other approaches may besubstituted therefor without departing from the inventive scope. Forexample, but not by way of limitation, other example approaches mayperform the operations remotely from the client device (e.g., at aserver).

FIG. 9 illustrates an example computing environment 900 with an examplecomputer device 905 suitable for use in some example implementations.Computing device 905 in computing environment 900 can include one ormore processing units, cores, or processors 910, memory 915 (e.g., RAM,ROM, and/or the like), internal storage 920 (e.g., magnetic, optical,solid state storage, and/or organic), and/or I/O interface 925, any ofwhich can be coupled on a communication mechanism or bus 930 forcommunicating information or embedded in the computing device 905.

Computing device 905 can be communicatively coupled to input/interface935 and output device/interface 940. Either one or both ofinput/interface 935 and output device/interface 940 can be a wired orwireless interface and can be detachable. Input/interface 935 mayinclude any device, component, sensor, or interface, physical orvirtual, which can be used to provide input (e.g., buttons, touch-screeninterface, keyboard, a pointing/cursor control, microphone, camera,braille, motion sensor, optical reader, and/or the like).

Output device/interface 940 may include a display, television, monitor,printer, speaker, braille, or the like. In some example implementations,input/interface 935 (e.g., user interface) and output device/interface940 can be embedded with, or physically coupled to, the computing device905. In other example implementations, other computing devices mayfunction as, or provide the functions of, an input/ interface 935 andoutput device/interface 940 for a computing device 905.

Examples of computing device 905 may include, but are not limited to,highly mobile devices (e.g., smartphones, devices in vehicles and othermachines, devices carried by humans and animals, and the like), mobiledevices (e.g., tablets, notebooks, laptops, personal computers, portabletelevisions, radios, and the like), and devices not designed formobility (e.g., desktop computers, server devices, other computers,information kiosks, televisions with one or more processors embeddedtherein and/or coupled thereto, radios, and the like).

Computing device 905 can be communicatively coupled (e.g., via I/Ointerface 925) to external storage 945 and network 950 for communicatingwith any number of networked components, devices, and systems, includingone or more computing devices of the same or different configuration.Computing device 905 or any connected computing device can befunctioning as, providing services of, or referred to as, a server,client, thin server, general machine, special-purpose machine, oranother label. For example but not by way of limitation, network 950 mayinclude the blockchain network, and/or the cloud.

I/O interface 925 can include, but is not limited to, wired and/orwireless interfaces using any communication or I/O protocols orstandards (e.g., Ethernet, 802.11xs, Universal System Bus, WiMAX, modem,a cellular network protocol, and the like) for communicating informationto and/or from at least all the connected components, devices, andnetwork in computing environment 900. Network 950 can be any network orcombination of networks (e.g., the Internet, local area network, widearea network, a telephonic network, a cellular network, satellitenetwork, and the like).

Computing device 905 can use and/or communicate using computer-usable orcomputer-readable media, including transitory media and non-transitorymedia. Transitory media includes transmission media (e.g., metal cables,fiber optics), signals, carrier waves, and the like. Non-transitorymedia includes magnetic media (e.g., disks and tapes), optical media(e.g., CD ROM, digital video disks, Blu-ray disks), solid state media(e.g., RAM, ROM, flash memory, solid-state storage), and othernon-volatile storage or memory.

Computing device 905 can be used to implement techniques, methods,applications, processes, or computer-executable instructions in someexample computing environments. Computer-executable instructions can beretrieved from transitory media, and stored on and retrieved fromnon-transitory media. The executable instructions can originate from oneor more of any programming, scripting, and machine languages (e.g., C,C++, C#, Java, Visual Basic, Python, Perl, JavaScript, and others).

Processor(s) 910 can execute under any operating system (OS) (notshown), in a native or virtual environment. One or more applications canbe deployed that include logic unit 955, application programminginterface (API) unit 960, input unit 965, output unit 970, learning andencoding unit 975, P and R determination unit 980, protocol and actionunit 985, and inter-unit communication mechanism 995 for the differentunits to communicate with each other, with the OS, and with otherapplications (not shown).

For example, the learning and encoding unit 975, the P and Rdetermination unit 980, and the protocol and action unit 985 mayimplement one or more processes shown above with respect to thestructures described above. The described units and elements can bevaried in design, function, configuration, or implementation and are notlimited to the descriptions provided.

In some example implementations, when information or an executioninstruction is received by API unit 960, it may be communicated to oneor more other units (e.g., logic unit 955, input unit 965, learning andencoding unit 975, P and R determination unit 980, and protocol andaction unit 985).

For example, the learning and encoding unit 975 may receive and processinformation, from one or more sensors, perform feature extraction, learnaction features, and the like, as explained above. An output of thelearning and encoding unit 975 is provided to the P and R determinationunit 980, which performs the necessary forecasting based on theapplication of the neural networks as described above and illustrated inFIG. 2, for example, and also determines R to provide a risk of fallingfor a subject in an environment, as also explained above. Additionally,the protocol and action unit 985 may provide a signal associated with anaction, based on the output of the learning and encoding unit 975 andthe P and R determination unit 980.

In some instances, the logic unit 955 may be configured to control theinformation flow among the units and direct the services provided by APIunit 960, input unit 965, learning and encoding unit 975, P and Rdetermination unit 980, and protocol and action unit 985 in some exampleimplementations described above. For example, the flow of one or moreprocesses or implementations may be controlled by logic unit 955 aloneor in conjunction with API unit 960.

FIG. 10 shows an example environment suitable for some exampleimplementations. Environment 1000 includes devices 1005-1045, and eachis communicatively connected to at least one other device via, forexample, network 1060 (e.g., by wired and/or wireless connections). Somedevices may be communicatively connected to one or more storage devices1030 and 1045.

An example of one or more devices 1005-1045 may be computing devices 905described in FIG. 9, respectively. Devices 1005-1045 may include, butare not limited to, a computer 1005 (e.g., a laptop computing device)having a monitor and an associated webcam as explained above, a mobiledevice 1010 (e.g., smartphone or tablet), a television 1015, a deviceassociated with a vehicle 1020, a server computer 1025, computingdevices 1035-1040, storage devices 1030 and 1045.

In some implementations, devices 1005-1020 may be considered userdevices associated with the users, who may be remotely obtaining asensed input used a inputs for the forecasting model and/or riskprofiling. In the present example implementations, one or more of theseuser devices may be associated with one or more cameras, that can senseinformation as needed for the predictive modeling and the riskprofiling, as explained above.

Aspects of the example implementations may have various advantages andbenefits. For example, but not by way of limitation, the present exampleimplementations provide predictive approaches and activity analysis.Additionally, the present example implementations take a passiveapproach, using sensors in the surroundings of the subject. Further, thepresent example implementations may distinguish falls from other dailyactivities, such as standing, sitting, lying down, walking, andascending or descending stairs.

Additionally, the example implementations herein are directed to acamera having death and motion sensors, and is capable of usinggrayscale input, without requiring RGB. Further, the exampleimplementations are not only directed to human movement in the bed, butin any safe position.

Thus, the example implementations may provide prediction and riskprofiling that permits high recall for high risk subjects, in accordancewith a rule-based protocol approach. While these example implementationsare primarily directed to a subject in a hospital setting, otherexamples may be considered. For example, but not by way of limitation,the foregoing example implementations may be applied in a non-hospitalsetting, such as in a residence of the subject, to address conditionssuch as sleepwalking or other abnormal movement in sleep. Further, theexample implementation may be incorporated with other types of sensorsthat may predict and assess risk with respect to other types ofconditions, such as sleep apnea, sudden infant syndrome (SIDS), or otherconditions. In a manner similar to that explained above, with respect tocontrol of a hospital bed, the example implementations may control othermedical devices, such as intravenous feed, oxygen supply, breathingsupport, etc.

Although a few example implementations have been shown and described,these example implementations are provided to convey the subject matterdescribed herein to people who are familiar with this field. It shouldbe understood that the subject matter described herein may beimplemented in various forms without being limited to the describedexample implementations. The subject matter described herein can bepracticed without those specifically defined or described matters orwith other or different elements or matters not described. It will beappreciated by those familiar with this field that changes may be madein these example implementations without departing from the subjectmatter described herein as defined in the appended claims and theirequivalents.

Aspects of certain non-limiting embodiments of the present disclosureaddress the features discussed above and/or other features not describedabove. However, aspects of the non-limiting embodiments are not requiredto address the above features, and aspects of the non-limitingembodiments of the present disclosure may not address features describedabove.

What is claimed is:
 1. A computer-implemented method, comprising: basedon information of a subject positioned on an object in an environment,generating a data stream; for the data stream, extracting featuresassociated with a movement of the subject with respect to the object orthe environment, wherein the movement is represented by spatio-temporalfeatures extracted from sensors; generating a prediction associated witha likelihood of the movement based on the extracted features, and a riskprofile of the movement based on a plurality of fall risk factors; andapplying the prediction and the risk profile to a rule base to performan action.
 2. The computer—implemented method of claim 1, wherein theinformation is sensed by at least one of a camera sensor and a motionsensor.
 3. The computer—implemented method of claim 2, wherein thecamera sensor comprises a depth sensor or a thermal sensor.
 4. Thecomputer—implemented method of claim 1, wherein the extracting thefeatures comprises applying the dense trajectories to extract thefeatures or Convolutional Neural Networks (CNN) to learn thespatio-temporal features from a sequence of frames in the data stream,applying an automatic learning system to learn the features, andencoding sequences of feature vectors based on the learned features andthe extracted features.
 5. The computer—implemented method of claim 1,wherein the generating the prediction comprises providing, over a timehorizon, the extracted features to a recurrent neural network or aconvolutional neural network to generate an output that is provided tothe dense layer, an output of which is fed to a softmax layer togenerate the prediction.
 6. The computer—implemented method of claim 1,wherein the risk profile is generated by receiving the plurality of fallrisk factors that comprise factors specific to the subject, factorsspecific to an environment associated with the subject, and factorsassociated with behavioral routines associated with the environment andthe subject, over time.
 7. The computer—implemented method of claim 1,wherein the rule base comprises determining whether the predictionexceeds a first threshold and whether the risk profile exceeds a secondthreshold, and based on the determining, performing the action.
 8. Thecomputer—implemented method of claim 1, wherein the action comprises oneor more of generating a signal for the subject, controlling a state ofthe object, and providing a command to an external resource to performthe action.
 9. The computer—implemented method of claim 1, wherein thesubject is a person, the object is a bed or a chair, and the movement isthe person falling from the bed or the chair.
 10. A non-transitorycomputer readable medium having a storage that stores instructions, theinstructions executed by a processor, the instructions comprising: basedon information of a subject positioned on an object in an environment,generating a data stream; for the data stream, extracting featuresassociated with a movement of the subject with respect to the object orthe environment, wherein the movement is represented by spatio-temporalvisual features; generating a prediction associated with a likelihood ofthe movement based on the extracted features, and a risk profile of themovement based on a plurality of fall risk factors; and applying theprediction and the risk profile to a rule base to perform an action. 11.The non-transitory computer readable medium of claim 10, wherein theinformation is sensed by at least one of a camera sensor and/or a motionsensor, and the camera sensor comprises a depth sensor or a thermalsensor.
 12. The non-transitory computer readable medium of claim 10,wherein the extracting the features comprises applying the densetrajectories or Convolutional Neural Networks (CNN) to learn thespatio-temporal features to extract the features from a sequence offrames in the data stream, applying an automatic learning system tolearn the features, and encoding sequences of feature vectors based onthe learned features and the extracted features.
 13. The non-transitorycomputer readable medium of claim 10, wherein the generating theprediction comprises providing, over a time horizon, the extractedfeatures to a recurrent neural network or a convolutional neural networkto generate an output that is provided to the dense layer, an output ofwhich is fed to a softmax layer to generate the prediction.
 14. Thenon-transitory computer readable medium of claim 10, wherein the riskprofile is generated by receiving the plurality of fall risk factorsthat comprise factors specific to the subject, factors specific to anenvironment associated with the subject, and factors associated withbehavioral routines associated with the environment and the subject,over time.
 15. The non-transitory computer readable medium of claim 10,wherein the rule base comprises determining whether the predictionexceeds a first threshold and whether the risk profile exceeds a secondthreshold, and based on the determining, performing the action.
 16. Thenon-transitory computer readable medium of claim 10, wherein the actioncomprises one or more of generating a signal for the subject,controlling a state of the object, and providing a command to anexternal resource to perform the action.
 17. The non-transitory computerreadable medium of claim 10, wherein the subject is a person, the objectis a bed or a chair, and the movement is the subject falling from thebed or the chair.
 18. A processor capable of processing a request, theprocessor configured to perform the operations of: based on informationof a subject positioned on a bed or chair in an environment, generatinga data stream; for the data stream, extracting features associated witha fall of the subject with respect to the bed or chair, or theenvironment, wherein the fall is represented by spatio-temporal visualfeatures; generating a prediction associated with a likelihood of thefall based on the extracted features, and a risk profile of the fallbased on a plurality of fall risk factors; and applying the predictionand the risk profile to a rule base to perform an action.
 19. Theprocessor of claim 18, wherein the extracting the features comprisesapplying the dense trajectories to extract the features or ConvolutionalNeural Networks (CNN) to learn the spatio-temporal features from asequence of frames in the data stream, applying an automatic learningsystem to learn the features, and encoding sequences of feature vectorsbased on the learned features and the extracted features, the generatingthe prediction comprises providing, over a time horizon, the extractedfeatures to a recurrent neural network or a deep neural network togenerate an output that is provided to the dense layer, an output ofwhich is fed to a softmax layer to generate the prediction, and the riskprofile is generated by receiving the plurality of fall risk factorsthat comprise factors specific to the subject, factors specific to anenvironment associated with the subject, and factors associated withbehavioral routines associated with the environment and the subject,over time.
 20. The processor of claim 18, wherein the rule basecomprises determining whether the prediction exceeds a first thresholdand whether the risk profile exceeds a second threshold, and based onthe determining, performing the action, and the action comprises one ormore of generating a signal for the subject, controlling a state of thebed or chair, and providing a command to an external resource to performthe action.